Imaging apparatus and method for controlling same

ABSTRACT

An imaging element includes a plurality of photoelectric conversion units that output an image signal for each pixel through a micro lens. An imaging signal processing circuit separates image signals output from the imaging element into a left-eye image signal and a right-eye image signal. An image combining circuit generates combined image data by performing arithmetic average processing for left-eye image data and right-eye image data. A recording medium control I/F unit controls to record left-eye image data and right-eye image data for use in 3D display and combined image data for use in 2D display in different regions in an image file.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/337,651 filed on Oct. 28, 2016, which is a continuation of U.S.patent application Ser. No. 13/615,996, filed on Sep. 14, 2012, now U.S.Pat. No. 9,521,395, which claims the benefit of and priority fromJapanese Patent Application No. 2011-220409, filed on Oct. 4, 2011, fromJapanese Patent Application No. 2011-264204, filed on Dec. 2, 2011, andfrom Japanese Patent Application No. 2011-264282, filed on Dec. 2, 2011,the disclosures of each of which are hereby incorporated by referenceherein in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an imaging apparatus and particularlyrelates to processing for recording image data for use inthree-dimensional display (hereinafter also referred to as “3D display”)and image data for use in two-dimensional display (hereinafter alsoreferred to as “2D display”) in a file.

Description of the Related Art

In recent years, there has been a rapid increase in the prevalence ofdevices associated with stereoscopic images such as three-dimensional(3D) cinema, 3D display, or the like. Conventionally, photographingstereoscopic images has been carried out by film cameras or the like.However, with the prevalence of digital imaging apparatuses,photographing original images for generating stereoscopic images usingdigital cameras, digital video cameras, or the like has become common.

As a mechanism by which a user views a stereoscopic image, data for aright-eye image and a left-eye image with parallax in the left-rightdirection is prepared so as to correspond to an image of the objectviewed with the left eye and an image of the object viewed with theright eye. A user can view stereoscopic images by viewing the right-eyeimage and the left-eye image with his/her right eye and left eye,respectively. Examples of such a method include a method for dividing aparallax image to be viewed, such as a parallax barrier method, alenticular method, or the like. Also, a method for providing differentimages to the left eye and the right eye of a user via a filter havingdifferent characteristics between the left and right sides thereof isknown.

On the other hand, as a method for capturing an image which is viewableas a stereoscopic image, Japanese Patent Laid-Open No. 58-24105discloses a method for simultaneously capturing images at differentviewpoints. Japanese Patent Laid-Open No. 58-24105 discloses asolid-state imaging element in which a plurality of micro lenses isformed and at least one pair of photodiodes is arranged close to each ofthe micro lenses. Of the pair of photodiodes, a first image signal isobtained from the output of one photodiode and a second image signal isobtained from the output of the other photodiode. A user can view astereoscopic image using the first and second image signals as aleft-eye image signal and a right-eye image signal, respectively.

Although left-eye image data and right-eye image data can be managed asseparate files, the following circumstance may occur for management ofimage data. In other words, the management of image data as separatefiles becomes complex as well as a stereoscopic image cannot be viewedif one of the files is lost. Accordingly, in the file generation methoddisclosed in Japanese Patent Laid-Open No. 2010-200354, left-eye imagedata and right-eye image data are recorded in the same file, and thus,efficient file management can be realized. In this case, a left-eyeimage or a right-eye image is used for two-dimensional display, whereasboth of a left-eye image and a right-eye image are used forthree-dimensional display.

Japanese Patent Laid-Open No. 2008-518317 discloses an output parallaxmap having an output element which has an output value corresponding toa shift to be applied to each pixel of a first image. A second image canbe generated based on the output parallax map and the first image. Thiscalculation is based on an input parallax map having input elements eachhaving an input value and can generate a second image using the inputimage and the parallax map in order to render a multi-view image.

Assume the case where photographing is performed using the solid-stateimaging element disclosed in Japanese Patent Laid-Open No. 58-24105 inwhich a plurality of micro lenses is formed and at least one pair ofphotodiodes is arranged close to each of the micro lenses. In thesolid-state imaging element, one of a pair of photodiodes outputs aleft-eye image signal which is obtained by photoelectrically convertinga light flux having passed through a region of an exit pupil of animaging optical system and the other outputs a right-eye image signalwhich is obtained by photoelectrically converting a light flux havingpassed through a region different from the aforementioned region of theexit pupil. In this case, depending on the type of an object, neitherthe left-eye image nor the right-eye image may be an image reflectingthe shape of the object.

For example, in a photographic scene in which light from a point lightsource is photographed in a blurred manner, a photograph of the lightsource blurred in a circular pattern should be taken originally.However, when an image is captured by the solid-state imaging elementdisclosed in Japanese Patent Laid-Open No. 58-24105, the captured imagemay be in a semicircular or elliptical shape not reflecting the shape ofan object. In addition, for example, the shape of the object which iscaptured as an image is photographed in a different way between theleft-eye image and the right-eye image so that the left half of theobject in the left-eye image is missing and the right half of the objectin the right-eye image is missing. The reason for this is that, amongthe light fluxes emitted from the exit pupil of the imaging opticalsystem, the region of light received by a photodiode is different alongthe optical axis serving as the boundary.

Even if such left-eye image data and right-eye image data are generatedas one file using the technique disclosed in Japanese Patent Laid-OpenNo. 2010-200354, the imaging apparatus may display a two-dimensionalimage having an incorrect shape upon image replay.

In addition, even if there is an attempt to calculate a second imageusing the technique disclosed in Japanese Patent Laid-Open No.2008-518317, an image reflecting the correct shape of an object cannotbe obtained. In other words, an original image is photographed in ashape different from that of the actual object, and thus, the calculatedsecond image cannot be reproduced as a correct image.

When a parallax map is used, an image file needs to be created inaccordance with a file format compatible with a generally-usedapplication. When a file including parallax map data and a fileincluding image data are created separately, a plurality of files needsto be managed. In consideration of the usability of file management by auser, it is preferable that both data are enclosed in a single file.

SUMMARY OF THE INVENTION

Accordingly, the present invention provides an imaging apparatus thatenhances reproducibility of a reproduction image by recording left-eyeimage data and right-eye image data for use in two-dimensional displayand image data reflecting the correct shape of an object for use intwo-dimensional display in the same file. Also, the present inventionprovides an imaging apparatus that is capable of reproducing an imagereflecting the correct shape of an object using the image data and theparallax map included in the image file even in the case of thetwo-dimensional display or the three-dimensional display.

According to an aspect of the present invention, an imaging apparatus isprovided that has an imaging element comprising a plurality ofphotoelectric conversion units configured to output a plurality of imagesignals by photoelectrically converting light fluxes each having passedthrough a different region of an exit pupil of an imaging optical systemand records data in an image file by processing the plurality of imagesignals by means of the imaging element. The imaging apparatus includesa signal processing unit configured to acquire left-eye image data andright-eye image data from the image signals output from the imagingelement and generate combined image data for two-dimensional display byadditively combining the left-eye image data and the right-eye imagedata; and a recording control unit configured to control to record theleft-eye image data, the right-eye image data, and the combined imagedata generated by the signal processing unit in different regions of theimage file.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram schematically illustrating the general configurationof an imaging element according to an embodiment of the presentinvention.

FIG. 2A is a diagram illustrating the configuration of one pixel of animaging element.

FIG. 2B is a diagram illustrating an exemplary configuration of aplurality of pixels of an imaging element.

FIG. 3 is a conceptual diagram illustrating how light fluxes emittedfrom the exit pupil of a photographing lens enter an imaging element.

FIG. 4 is a diagram illustrating an exemplary configuration of animaging apparatus according to an embodiment of the present invention.

FIG. 5 is a diagram illustrating an example of a file structureaccording to a first embodiment of the present invention.

FIG. 6 is a flowchart illustrating an example of file generationprocessing according to the first embodiment of the present invention.

FIG. 7 is a flowchart illustrating another example of file generationprocessing according to the first embodiment of the present invention.

FIG. 8 is a diagram illustrating another example of a file structureaccording to the first embodiment of the present invention.

FIG. 9 is a diagram illustrating another example of a file structureaccording to the first embodiment of the present invention.

FIG. 10 is a diagram illustrating an exemplary configuration of animaging apparatus according to a second embodiment of the presentinvention.

FIGS. 11A to 11C are diagrams illustrating an example of parallax mapgeneration processing.

FIG. 12 is a diagram schematically illustrating a parallax map.

FIG. 13 is a diagram illustrating an example of a typical DCF image filestructure.

FIG. 14 is a flowchart illustrating processing from the start ofphotographing to the recording of an image file.

FIG. 15 is a flowchart illustrating image file creation processing.

FIG. 16 is a flowchart illustrating parallax map generation processing.

FIG. 17 is a diagram illustrating another image file structure.

FIG. 18 is a flowchart illustrating parallax map generation processing.

FIG. 19 is a diagram illustrating the file structure of an image fileincluding RAW image data.

FIG. 20 is a flowchart illustrating parallax map generation processing

FIG. 21 is a diagram illustrating an example of a typical DCF image filestructure.

FIG. 22 is a diagram illustrating an image file having left-eye imagedata and right-eye image data.

FIG. 23 is a diagram illustrating an image file having a parallax map.

FIG. 24 is a flowchart illustrating DCF image file reproductionprocessing.

FIG. 25 is a flowchart illustrating 2D display image reproductionprocessing.

FIG. 26 is a flowchart illustrating 3D display image reproductionprocessing.

FIG. 27 is a flowchart illustrating left/right image 3D reproductionprocessing.

FIG. 28 is a flowchart illustrating parallax map 3D reproductionprocessing.

DESCRIPTION OF THE EMBODIMENTS

FIG. 1 is a diagram schematically illustrating an exemplaryconfiguration of an imaging element that is applied to the imagingapparatus according to one embodiment of the present invention. Animaging element 100 includes a pixel array 101, a vertical selectioncircuit 102 that selects a row in the pixel array 101, and a horizontalselection circuit 104 that selects a column in the pixel array 101. Aread-out circuit 103 reads a signal of a pixel which has been selectedfrom the pixels in the pixel array 101 by the vertical selection circuit102. The read-out circuit 103 has a memory for accumulating signals, again amplifier, an AD converter, or the like for each column.

A serial interface (SI) unit 105 determines the operation mode of eachcircuit in accordance with the instructions given by an externalcircuit. The vertical selection circuit 102 sequentially selects aplurality of rows of the pixel array 101 so that a pixel signal(s) isextracted to the read-out circuit 103. Also, the horizontal selectioncircuit 104 sequentially selects a plurality of pixel signals read bythe read-out circuit 103 for each row. Note that the imaging element 100includes a timing generator that provides a timing signal to thevertical selection circuit 102, the horizontal selection circuit 104,the read-out circuit 103, and the like, a control circuit, and the likein addition to the components shown in FIG. 1, but no detaileddescription thereof will be given.

FIGS. 2A and 2B are diagrams illustrating an exemplary configuration ofa pixel of the imaging element 100. FIG. 2A schematically shows theconfiguration of one pixel. FIG. 2B shows the arrangement of the pixelarray 101. A pixel 201 shown in FIG. 2A has a micro lens 202 serving asan optical element and a plurality of photodiodes (hereinafterabbreviated as “PD”) serving as light receiving elements.

Although FIG. 2A shows an example in which a left-side PD 203 and aright-side PD 204 are provided for one pixel, three or more (e.g., fouror nine) PDs may also be used. The PD 203 photoelectrically converts thereceived light flux to thereby output a left-eye image signal. The PD204 photoelectrically converts the received light flux to thereby outputa right-eye image signal. Note that the pixel 201 also includes a pixelamplifier for extracting a PD signal to the read-out circuit 103, a rowselection switch, a reset switch for resetting a PD signal, and the likein addition to the components shown in FIG. 2A.

In order to provide a two-dimensional image, the pixel array 101 isarranged in a two-dimensional array such as a plurality of pixels 301,302, 303, and 304 as shown in FIG. 2B. Each of PDs 301L, 302L, 303L, and304L corresponds to the PD 203 shown in FIG. 2A. Also, each of PDs 301R,302R, 303R, and 304R corresponds to the PD 204 shown in FIG. 2A. Inother words, the imaging element for use in the present embodimentincludes a plurality of pixels each having a first photoelectricconversion unit (PD 203) configured to output a left-eye image signaland a second photoelectric conversion unit (PD 204) configured to outputa right-eye image signal.

Next, a description will be given of the light receiving of the imagingelement 100 having the pixel configuration shown in FIG. 2B. FIG. 3 is aconceptual diagram illustrating how light fluxes emitted from the exitpupil of a photographing lens enter the imaging element 100. The pixelarray 101 has a micro lens 202, a color filter 403, and PDs 404 and 405.The PDs 404 and 405 correspond to the PDs 203 and 204 shown in FIG. 2A,respectively.

In FIG. 3, the center axis of the light flux emitted from an exit pupil406 of a photographing lens to the micro lens 202 is an optical axis409. The light emitted from the exit pupil 406 enters the imagingelement 100 centered on the optical axis 409. Each of the partialregions 407 and 408 is a region of the exit pupil 406 of thephotographing lens. Light beams 410 and 411 are the outermost peripherallight beams of light passing through the partial region 407. Light beams412 and 413 are the outermost peripheral light beams of light passingthrough the partial region 408.

Among the light fluxes emitted from the exit pupil 406, the upper lightflux enters the PD 405 and the lower light flux enters the PD 404, withthe optical axis 409 serving as the boundary. In other words, each ofthe PDs 404 and 405 receives a light flux emitted from a differentregion of the exit pupil of the photographing optical system. In thismanner, each of the light receiving elements (PDs 404 and 405) detectslight that has passed through a different region of the exit pupil.Thus, in the case where light from a point light source is photographedin a blurred manner, each of the light receiving elements obtains aphotographed image with a different shape.

FIG. 4 is a diagram illustrating an exemplary configuration of animaging apparatus of the present embodiment. With reference to FIG. 4, adescription will be given of an exemplary application of the imagingelement 100 shown in FIG. 1 to a digital camera. A lens unit 501constituting the imaging optical system focuses the light reflected froman object on an imaging element 505. The imaging element 505 correspondsto the imaging element 100 shown in FIG. 1 and has the pixelconfiguration shown in FIG. 2B. A lens drive device 502 executes zoomcontrol, focus control, diaphragm control, or the like. A mechanicalshutter 503 is controlled by a shutter drive device 504. The imagingelement 505 converts an object image focused by the lens unit 501 intoan image signal. An imaging signal processing circuit 506 performsvarious kinds of processing (pixel interpolation processing, colorconversion processing, or the like) or correction on the image signaloutput by the imaging element 505. A timing generation unit 507 outputsa timing signal required for the imaging element 505 or the imagingsignal processing circuit 506.

A system control unit 509 is a control unit that performs variouscomputations and controls the imaging apparatus overall. A CPU (CentralProcessing Unit) (not shown) provided therein interprets and executes aprogram to thereby perform various kinds of processing. Note that thesystem control unit 509 can perform AF (Auto Focus) control by detectinga phase difference between left-eye image data and right-eye image data.

A memory unit 508 includes a memory that temporarily stores image data.A recording medium control interface unit (hereinafter abbreviated as“I/F unit”) 510 which performs recording control is provided forrecording/reading image data or the like in/from a recording medium 511.The recording medium 511 which is detachable from the imaging apparatusis a semiconductor memory or the like. An external I/F unit 512transmits/receives data to/from an external device such as a computerdevice or the like so that a user can process an image. Also, a user canoperate a digital camera using an operation unit (not shown) connectedto the external I/F unit 512.

The image combining circuit 513 performs arithmetic average processingfor image data output from the imaging element 505 for each pixel andperforms processing for acquiring only the required PD signals. In theexample shown in FIG. 2, for the pixel signals 301L, 301R, 302L, 302R,303L, 303R, 304L, and 304R output from the PDs, the pixel signalsextracted from the outputs of the left-side PDs only are 301L, 302L,303L, and 304L, whereas the pixel signals extracted from the output ofthe right-side PDs only are 301R, 302R, 303R, and 304R. Combined imagedata is created from ((301L+301R)/2, (302L+302R)/2, (303L+303R)/2, and(304L+304R)/2) which are obtained by arithmetically averaging theoutputs of the left-side PDs and the right-side PDs. Note that onlyaddition processing may be performed during addition combinationprocessing and division processing and adjustment of the dynamic rangemay be performed during post processing.

A compression/decompression circuit 515 compresses image data stored inthe memory unit 508 in accordance with a predetermined image compressionmethod (e.g., adaptive discrete cosine transform (ADCT)). Also, thecompression/decompression circuit 515 has a function of writing thecompressed image data to the memory unit 508 and a function ofdecompressing image data read out from the memory unit 508 and writingthe decompressed image data to the memory unit 508. A display unit 516displays various pieces of information and a photographed image inaccordance with display data from a display control circuit 517. Theimaging signal processing circuit 506 performs image processing byallocating imaging data output from the imaging element 505 to therespective data for a left-eye image and a right-eye image. The memoryunit 508 stores the output data output from the imaging signalprocessing circuit 506, the combined image data generated by an imagecombining circuit 513, and the like.

Next, a description will be given of the operation of a digital cameraduring photographing. When the main power supply of an imaging apparatusis turned ON, the power supply of a control system circuit unit isturned ON and the power supply of an imaging processing system circuitsuch as the imaging signal processing circuit 506 is also turned ON.When a user operates a release button (not shown), the system controlunit 509 computes focus state detection based on data from the imagingelement 505 to thereby calculate the distance between the imagingapparatus and the object. Then, the lens drive device 502 drives themovable lens of the lens unit 501 and the system control unit 509determines whether or not the focus state is in-focus.

When the system control unit 509 determines that the focus state is notin-focus, the system control unit 509 controls the drive of the lensunit 501 again to thereby execute focus state detection processing. Forcomputation of the distance between the imaging apparatus and theobject, besides a method for calculating the distance from data obtainedby the imaging element 505, a method for computing the distance using adistance measuring dedicated device (not shown) may also be used. Thesystem control unit 509 starts the photographing operation afterdetermination that the focus state is in-focus. When the photographingoperation has been completed, the imaging signal processing circuit 506processes the image signal output from the imaging element 505, and thesystem control unit 509 controls the writing of the image data to thememory unit 508.

Imaging data output from the imaging element 505 is output as imagesignals from a plurality of PDs. In the example shown in FIG. 2B, imagesignals are output in the order of the PDs 301L, 301R, 302L, 302R, 303L,303R, 304L, and 304R. The imaging signal processing circuit 506 performsimage processing by allocating imaging data output from the imagingelement 505 to left-eye image data and right-eye image data. Left eyeimage data is image data obtained as a result of selecting andprocessing only the output from the left-side PDs 301L, 302L, 303L, and304L shown in FIG. 2B. Also, right-eye image data is image data obtainedas a result of selecting and processing only the output from theright-side PDs 301R, 302R, 303R, and 304R shown in FIG. 2B. Left eyeimage data and right-eye image data are separately held in the memoryunit 508.

The image combining unit 513 reads the respective data for a left-eyeimage and a right-eye image held in the memory unit 508 to therebygenerate a combined image data. The generated combined image data isstored in the memory unit 508. Image processing executed by the imagecombining unit 513 is processing for calculating an arithmetic averagevalue for each pixel of a left-eye image and a right-eye image. Thus,the combined image generated by image processing has a shape reflectingthe shape of an object. In other words, even when an object isphotographed with the imaging element 505 in a state where the shape ofthe object is different between a left-eye image and a right-eye image,the shape of the object image is interpolated by image processingperformed by the image combining unit 513, resulting in the generationof image data in a correct shape. For example, if an object has acircular shape and both a left-eye image and a right-eye image do nothave a circular shape, the combined image has the same circular shape asthat of the object.

In the exemplary configuration shown in FIG. 4, the imaging signalprocessing circuit 506 performs processing for dividing image dataobtained by the imaging element 505 into left-eye image data andright-eye image data and the image combining circuit 513 performsprocessing for combining both image data. The present invention is notlimited thereto, but the imaging signal processing circuit 506 may alsobe adapted to combine left-eye image data with right-eye image data bothsubjected to image processing.

The memory unit 508 accumulates left-eye image data, right-eye imagedata, and combined image data generated by the image combining circuit513. The system control unit 509 controls to record data in a detachablerecording medium 511 such as a semiconductor memory or the like via arecording medium control I/F unit 510. Alternatively, a user can performimage processing by outputting image data to an external device such asa computer via the external I/F unit 512.

First Embodiment

Hereinafter, a description will be given of a first embodiment of thepresent invention. In the first embodiment, a system configuration isprovided that is capable of outputting combined image data, right-eyeimage data, and left-eye image data when RAW image data for the left andright PDs, i.e., all of the PDs is input to the imaging signalprocessing circuit 506. The term “RAW image” means an image obtainedfrom data for each of the pixels of the imaging element 505, i.e., animage prior to image processing such as development processing or thelike.

FIG. 5 is a diagram illustrating an exemplary file structure of imagedata according to the present embodiment. An image data file 600 is aDCF image data file that is generally used for a digital camera. Theterm “DCF” (Design rule for Camera File system) refers to an image fileformat for handling image data in a digital camera using a commonspecification. The image data file 600 has the regions including a DCFheader section 610, a thumbnail image section 620, and a JPEG imagesection 630, where JPEG is an abbreviation for “Joint PhotographicExperts Group”.

The DCF header section 610 is a region, in which a predetermined datasize is given in advance, for storing DCF header information. The DCFheader information includes the following supplementary information andoffset information for specifying a region for storing each image data(The number in parentheses denotes a reference numeral).

Metadata A (611): photographing information, parameters, and the likeregarding image data stored in the JPEG image section 630.

Offset value B (612): offset data corresponding to the head position ofa combined thumbnail image section 621.

Offset value C (613): offset data corresponding to the head position ofa left-eye thumbnail image section 622.

Offset value D (614): offset data corresponding to the head position ofa right-eye thumbnail image section 623.

Offset value E (615): offset data corresponding to the head position ofa combined JPEG image section 631.

Offset value F (616): offset data corresponding to the head position ofa left-eye JPEG image section 632.

Offset value G (617): offset data corresponding to the head position ofa right-eye JPEG image section 633.

Each of the offset values B, C, D, E, F, and G is relative positioninformation between the reference position of the DCF header section 610and the calculated position of each image section, and the startingposition of each image data stored in an image file is specified usingeach of these values.

The thumbnail image section 620 is a region for storing thumbnail imagedata which is resized by thinning JPEG image data or the like stored inthe JPEG image section 630. Thumbnail image data is used, for example,when a plurality of reduced images is displayed on the screen of thedisplay unit 516 (when index display is performed). Thumbnail image datafor use in 2D display is recorded in the combined thumbnail imagesection 621. Thumbnail left-eye image data and thumbnail right-eye imagedata for use in 3D display are recorded in the left-eye thumbnail imagesection 622 and the right-eye thumbnail image section 623, respectively.

The JPEG image section 630 is a region for storing JPEG image data whichis obtained by compressing RAW image data subjected to image processingby the compression/decompression circuit 515. The image data is datawhich can be handled by many generally-used applications. JPEG imagedata for use in 2D display is recorded in the combined JPEG imagesection 631. JPEG left-eye image data and JPEG right-eye image data foruse in 3D display are recorded in the left-eye JPEG image section 632and the right-eye JPEG image section 633, respectively. Theaforementioned file structure is merely an example, and the filestructure in which the arrangement order of image sections is changeddepending on various applications such as image editing or the like mayalso be used.

FIG. 6 is a basic flowchart illustrating DCF file generation processingperformed by a digital camera. When the process starts in step S700, RAWdata for all of the combined PDs of the right-side PDs and the left-sidePDs, which has been read out from the imaging element 505, is expandedon the memory unit 508 in step S701, and the process advances to stepS702. In step S702, the imaging signal processing circuit 506 performsimage processing for RAW image data on the memory unit 508 using thedevelopment parameters for a RAW image, and the output image data isarranged on the memory unit 508. Then, the process advances to stepS703. Since the signals read out from all of the PDs are used for a RAWimage, the RAW image for an object can be recognized as image datareflecting the correct shape of the object even when light from theaforementioned point light source is photographed in a blurred manner.For convenience, an image generated by addition combination processingusing all of the combined PDs of the right-side PDs and the left-sidePDs is referred to as a “combined image”. A combined image for a RAWimage is referred to as a “combined RAW image”, a combined image for aJPEG image is referred to as a “combined JPEG image”, and a combinedimage for a thumbnail image is referred to as a “combined thumbnailimage”. Note that addition combination processing is intended to includearithmetic average processing.

In step S703, the combined RAW image data obtained in step S702 is inputto the imaging signal processing circuit 506, and development processingfor the combined RAW image data is performed using the developmentparameters for a JPEG image. Furthermore, combined JPEG image data andcombined thumbnail image data are generated by a predetermined imagecompression, and the process advances to step S704. It is determined instep S704 whether a 2D image file or a 3D image file is generated. Whena 3D image file is generated, the process advances to step S705, whereaswhen a 2D image file is generated, the process advances to step S709.Note that the term “3D image file” defined herein refers to a file in aformat which can handle both an image for 3D display and an image for 2Ddisplay.

In step S705, RAW data for all of the PDs obtained in step S701 is inputto the imaging signal processing circuit 506, and RAW image datacorresponding to the right-side PDs is extracted. The imaging signalprocessing circuit 506 performs image processing for the extracted RAWimage data using the development parameters for a RAW image and outputsright-eye RAW image data to the memory unit 508. Then, the processadvances to step S706. In step S706, the right-eye RAW image dataexpanded on the memory unit 508 is input to the imaging signalprocessing circuit 506, and image processing for the right-eye RAW imagedata is performed using the development parameters for a JPEG image.Furthermore, right-eye JPEG image data and right-eye thumbnail imagedata are generated by a predetermined image compression, and the processadvances to step S707.

In step S707, RAW data for all of the PDs obtained in step S701 is inputto the imaging signal processing circuit 506, and RAW image datacorresponding to the left-side PDs is extracted. The imaging signalprocessing circuit 506 performs image processing for the extracted RAWimage data using the development parameters for a RAW image and outputsleft-eye RAW image data to the memory unit 508. Then, the processadvances to step S708. In step S708, the left-eye RAW image dataexpanded on the memory unit 508 is input to the imaging signalprocessing circuit 506, and development processing for the left-eye RAWimage data is performed using the development parameters for a JPEGimage. Furthermore, left-eye JPEG image data and left-eye thumbnailimage data are generated by a predetermined image compression, and theprocess advances to step S709.

In step S709, information indicating a shutter speed, an exposure time,and the like during photographing is set as metadata A (see referencenumeral 611 shown in FIG. 5). Next, the offset values B to G (seereference numerals 612 to 617 shown in FIG. 5) corresponding to theimage sections are calculated on the basis of the data sizes of theimages generated until step S709, and these values are set. In thismanner, data for the DCF header section 610, including the metadata Aand the offset values B to G, is created on the memory unit 508, and theprocess advances to step S710. In step S710, data for the DCF headersection 610 created in step S709, data for the JPEG images, and data forthe thumbnail images are coupled as one DCF file having the structuredescribed in FIG. 5, and DCF file generation processing is ended.

On the other hand, if it is determined in step S704 that a 2D image fileis generated, the process advances to step S709. In step S709, metadataA (see reference numeral 611 shown in FIG. 5) regarding photographinginformation, parameters, and the like is set and the offset values B(see reference numeral 612) and E (see reference numeral 615)corresponding to the combined images are calculated and set. Here, apredetermined value (e.g., “0”) indicating that an image is not includedin the image data file 600 is set for the offset values C (see referencenumeral 613) and F (see reference numeral 616) corresponding toright-eye images and the offset values D (see reference numeral 614) andG (see reference numeral 617) corresponding to left-eye images.Alternatively, when a method for not recording the items of the offsetvalues is employed, the DCF header section 610 not having regionscorresponding to the items is created, and the process advances to stepS710. In step S710, a DCF file is generated from data for the DCF headersection 610, combined JPEG image data, and combined thumbnail imagedata, and a series of DCF file generation processing is ended. Thestructure of a 2D display file is the same as that of a conventional DCFfile, and thus, a detailed description thereof will be omitted.

In the first embodiment, right-eye image data, left-eye image data, andcombined image data by additively combining these image data aregenerated from the outputs of all of the PDs in the imaging element, andthus, image data reflecting the correct shape of an object can berecorded in a file as a two-dimensional display image. In other words, afile including not only left-eye image data and right-eye image datawhere an object is viewed from different directions but also combinedimage data for reproducing the correct shape of the object by additivelycombining these image data is obtained.

Variant Example of First Embodiment

Next, a description will be given of a variant example of the firstembodiment of the present invention. In the variant example, RAW datafor the right-side PDs and the left-side PDs is input to the imagingsignal processing circuit 506, and thus, right-eye image data andleft-eye image data are output. The right-eye image data and theleft-eye image data are output as combined image data by the imagecombining circuit 513. The file structure in the variant example is thesame as that of the first embodiment (see FIG. 5).

FIG. 7 is a basic flowchart illustrating DCF file generation processingaccording to a variant example of the first embodiment. When the processstarts in step S800, RAW data for the right-side PDs and the left-sidePDs, which has been read out from the imaging element 505, is expandedon the memory unit 508 in step S801, and the process advances to stepS802. In step S802, RAW data for the right-side PDs on the memory unit508 is extracted, and the imaging signal processing circuit 506 performsimage processing using the development parameters for a RAW image. Theoutput right-eye RAW image data is arranged on the memory unit 508, andthe process advances to step S803. In step S803, RAW data for theleft-side PDs on the memory unit 508 is extracted, and the imagingsignal processing circuit 506 performs image processing using thedevelopment parameters for a RAW image. The output left-eye RAW imagedata is arranged on the memory unit 508, and the process advances tostep S804.

In step S804, right-eye RAW image data and left-eye RAW image data areinput to the image combining circuit 513. The image combining circuit513 performs arithmetic average processing for both RAW image data foreach pixel to thereby generate combined RAW image data, and thegenerated combined RAW image data is arranged on the memory unit 508.Then, the process advances to step S805. Since interpolation processingrelating to the shape of an object is performed for image data subjectedto addition combination processing, the RAW image data for the objectcan be recognized as image data reflecting the correct shape of theobject even when light from the aforementioned point light source isphotographed in a blurred manner. In step S805, the combined RAW imagedata obtained in step S804 is input to the imaging signal processingcircuit 506, and development processing for the combined RAW image datais performed using the development parameters for a JPEG image.Furthermore, combined JPEG image data and combined thumbnail image dataare generated by a predetermined image compression, and the processadvances to step S806.

It is determined in step S806 whether a 2D image file or a 3D image fileis generated. When a 3D image file is generated, the process advances tostep S807, whereas when a 2D image file is generated, the processadvances to step S809. In step S807, the right-eye RAW image dataobtained in step S802 is input to the imaging signal processing circuit506, and development processing for the right-eye RAW image data isperformed using the development parameters for a JPEG image.Furthermore, right-eye JPEG image data and right-eye thumbnail imagedata are generated by a predetermined image compression, and the processadvances to step S808.

In step S808, the left-eye RAW image data obtained in step S803 is inputto the imaging signal processing circuit 506, and development processingfor the left-eye RAW image data is performed using the developmentparameters for a JPEG image. Furthermore, left-eye JPEG image data andleft-eye thumbnail image data are generated by a predetermined imagecompression, and the process advances to step S809. In step S809,information indicating a shutter speed, an exposure time, and the likeduring photographing is set as metadata A (see reference numeral 611shown in FIG. 5). Next, the offset values B to G (see reference numerals612 to 617 shown in FIG. 5) corresponding to the image sections arecalculated on the basis of the data sizes of the images generated untilstep S809, and these values are set. In this manner, data for the DCFheader section 610, including the metadata A and the offset values B toG for photographing information, is created on the memory unit 508, andthe process advances to step S810. In step S810, data for the DCF headersection 610 created in step S809, data for the JPEG images, and data forthe thumbnail images are coupled as one DCF file having the structuredescribed in FIG. 5, and DCF file generation processing is ended.

On the other hand, if it is determined in step S806 that a 2D image fileis generated, the process advances to step S809. In step S809, metadataA (see reference numeral 611 shown in FIG. 5) regarding photographinginformation, parameters, and the like is set and the offset values B(see reference numeral 612) and E (see reference numeral 615)corresponding to the combined images are calculated and set. Here, apredetermined value indicating that an image is not included in theimage data file 600 is set to the offset values C (see reference numeral613), D (see reference numeral 614), F (see reference numeral 616), andG (see reference numeral 617). Alternatively, the DCF header section 610is created without recording the items of the offset values in the imagefile, and the process advances to step S810.

As described above, in the variant example, right-eye image data andleft-eye image data are generated by the right-side PDs and theleft-side PDs, respectively, and combined image data is generated on thebasis of both image data. Consequently, image data reflecting thecorrect shape of an object can be recorded in a file as atwo-dimensional display image.

Another Variant Example of First Embodiment

Next, a description will be given of another variant example of thefirst embodiment of the present invention. In the variant example,combined image data, right-eye image data, left-eye image data, and RAWimage data are enclosed in the same file. FIG. 8 is a diagramillustrating an exemplary file structure of image data according toanother variant example. An image data file 900 consists of the regionsincluding a RAW header section 910, a thumbnail image section 920, adisplay JPEG image section 930, and a RAW image section 940.

The RAW header section 910 is a region, in which a predetermined datasize is given in advance, for storing RAW header information. RAW headerinformation includes the following supplementary information and offsetinformation for specifying a region for storing each image data (Thenumber in parentheses denotes a reference numeral).

Metadata A (911): photographing information, parameters, and the likeregarding image data stored in the RAW image section 940.

Offset value B (912): offset data corresponding to the head position ofa combined thumbnail image section 921.

Offset value C (913): offset data corresponding to the head position ofa left-eye thumbnail image section 922.

Offset value D (914): offset data corresponding to the head position ofa right-eye thumbnail image section 923.

Offset value E (915): offset data corresponding to the head position ofa 2D display JPEG image section 931.

Offset value F (916): offset data corresponding to the head position ofa left-eye display JPEG image section 932.

Offset value G (917): offset data corresponding to the head position ofa right-eye display JPEG image section 933.

Offset value H (918): offset data corresponding to the head position ofa left-eye RAW image section 941.

Offset value I (919): offset data corresponding to the head position ofa right-eye RAW image section 942.

The starting position of each image data stored in an image file isspecified using each of the offset values B, C, D, E, F, G, H, and I.

The thumbnail image section 920 is a region for storing thumbnail imagedata which is resized by thinning a JPEG image or the like stored in thedisplay JPEG image section 930. Thumbnail image data is used, forexample, when a plurality of reduced images is displayed on the imagedisplay section of an external device such as a personal computer or thelike (when index display is performed). Thumbnail image data for use in2D display is recorded in the combined thumbnail image section 921.Thumbnail left-eye image data and thumbnail right-eye image data for usein 3D display are recorded in the left-eye thumbnail image section 922and the right-eye thumbnail image section 923, respectively.

The display JPEG image section 930 is a region for storing JPEG imagedata which is obtained by compressing RAW image data stored in the RAWimage section 940 in a predetermined manner. The JPEG image data istypically used as a reference for the confirmation of images and can behandled by many applications. JPEG image data for use in 2D display isrecorded in the 2D display JPEG image section 931. Display JPEG left-eyeimage data and display JPEG right-eye image data for use in 3D displayare recorded in the left-eye display JPEG image section 932 and theright-eye display JPEG image section 933, respectively.

The RAW image section 940 is a region for storing RAW image data whichis output from the imaging element 505 without degradation. Since theRAW image data can be subjected to image reproduction processing(development) by an external device such as a personal computer or thelike, a high-quality print or high-level image editing that meets theobjectives of a user can be performed. In contrast, since a RAW imagehas a large amount of data and is not suitable for image display or thelike, image data in the display JPEG image section 930 is used uponactual 3D display. Thus, only RAW data may be recorded as material datain the RAW image section 940 in order to suppress the amount of data. Inthe variant example, RAW image data obtained from the left-side PDs isrecorded in the left-eye RAW image section 941, and RAW image dataobtained from the right-side PDs is recorded in the right-eye RAW imagesection 942.

In the case of the file structure shown in FIG. 9 as another example,RAW image data obtained from all of the PDs, which is not divided intoright-eye image data and left-eye image data, is recorded in the RAWimage section 940. In other words, the RAW image data recorded in theRAW image section 940 is selected and then processed into image data for3D display. For example, an external device such as a personal computeror the like acquires RAW image data from the RAW image section 940,separates the acquired RAW image data into right-eye image data andleft-eye image data, and performs development processing for theseparated image data to thereby generate image data for 3D display. Inthis case, the offset values B to H are obtained and the offset value His offset data corresponding to the head position of the RAW imagesection 940.

Next, a description will be given of RAW file generation processingaccording to the present embodiment. The RAW file generation processingis performed in the same manner as the DCF file generation processingdescribed in the first embodiment and the variant example of the firstembodiment. In other words, in accordance with the processing describedin FIG. 6 and FIG. 7, image data stored in the image sections 921, 922,and 923, the image sections 931, 932, and 933, the image sections 941and 942, or the image section 940 are arranged on the memory unit 508.Then, the metadata A such as photographing information, parameters, andthe like (see reference numeral 911) is created, and the offset values Bto I of the image sections (see reference numerals 912 to 919) arecalculated and set, whereby the RAW header section 910 is created. Then,processing for coupling the RAW header section 910 and image data as oneimage file having the file structure shown in FIG. 8 or FIG. 9 isexecuted, and file generation processing is ended. As described above,in the present embodiment, a file including image data reflecting thecorrect shape of an object as image data for 2D display and RAW imagedata prior to image processing can be generated.

Second Embodiment

Hereinafter, a description will be given of a second embodiment of thepresent invention. FIG. 10 is a diagram illustrating an exemplaryconfiguration of an imaging apparatus according to the second embodimentof the present invention. The imaging apparatus of the second embodimentis the same as the imaging apparatus of the first embodiment except thatthe imaging apparatus of the second embodiment includes a parallax mapgeneration circuit 514. The functions of other processing units are thesame as those described in the first embodiment, and thus, the followingdescription will be given by focusing on the parallax map generationcircuit 514. In the second embodiment, the imaging apparatus generates aparallax map and creates an image file enclosing the map. The parallaxmap generation circuit 514 calculates the positional shift amountbetween the left-eye image and the right-eye image on the basis of theposition of the object image in the combined image as a parallax amount.Then, information regarding the calculated parallax amount is recordedas a parallax map in the memory unit 508.

Next, a description will be given of the operation of a digital cameraduring photographing. As described in the first embodiment, when a useroperates a release button (not shown), the system control unit 509calculates the distance between the imaging apparatus and the object tothereby determine whether or not the focus state is in-focus. When thefocus state is in-focus, the imaging element 505 outputs image signalsand the imaging signal processing circuit 506 performs image processingby allocating the output image signals to left-eye image data andright-eye image data. Left eye image data and right-eye image data areseparately held in the memory unit 508. The image combining unit 513reads out the left-eye image data and the right-eye image data held inthe memory unit 508 to thereby generate combined image data. Thegenerated combined image data is stored in the memory unit 508. Imageprocessing executed by the image combining unit 513 is processing forcalculating an arithmetic average value for each pixel of a left-eyeimage and a right-eye image. Thus, the combined image generated by imageprocessing has a shape reflecting the shape of an object. In otherwords, even when an object is photographed with the imaging element 505in a state where the shape of the object is different between a left-eyeimage and a right-eye image, the shape of the object image isinterpolated by image processing performed by the image combining unit513, resulting in the generation of image data in a correct shape. Forexample, if an object has a circular shape and both a left-eye image anda right-eye image do not have a circular shape, the combined image hasthe same circular shape as that of the object.

The parallax map generation circuit 514 generates a parallax map andstores the parallax map data in the memory unit 508. The parallax mapgeneration circuit 514 generates a parallax map by utilizing thepositional shift amount between the left-eye image and the right-eyeimage on the basis of the position of the combined image as a parallaxamount.

FIGS. 11A to 11C are diagrams illustrating an example of parallax mapgeneration processing. Reference numeral 601 shown in FIG. 11A denotesthe composition of the image obtained by photographing objects.Reference numerals 602, 603, and 604 denote objects. In the compositionshown in FIG. 11A, the objects 602, 603, and 604 are arrayed in orderfrom the top to the bottom. Also, as shown in FIG. 11C, the objects arearranged in a line in the depth direction. Reference numeral 604 denotesthe closest object and reference numeral 602 denotes the furthestobject.

FIG. 11B shows a stereo image obtained by photographing the compositionshown in FIG. 11A. An image 605 is a left-eye image and an image 606 isa right-eye image. In the left-eye image 605, the objects 602, 603, and604 are denoted as 607L, 608L, and 609L, respectively. In the right-eyeimage 606, the objects 602, 603, and 604 are denoted as 607R, 608R, and609R, respectively.

There is a positional shift between an object image in the left-eyeimage 605 and the object image in the right-eye image 606. In thepresent embodiment, the amount of positional shift between two images isdefined as a parallax amount. Reference numeral 610 denotes the amountof shift in position of the object 602 in the right-eye image 606relative to the object 602 in the left-eye image 605 as a reference,i.e., a parallax amount between 607L and 607R. Likewise, referencenumeral 611 denotes the amount of shift in position of the object 604 inthe right-eye image 606 relative to the object 604 in the left-eye image605 as a reference, i.e., a parallax amount between 609L and 609R. Theposition of the object 603 in the left-eye image 605 is the same as thatin the right-eye image 606. In other words, there is no parallax amountfor the object 603.

Firstly, the parallax map generation circuit 514 detects objectsincluded in the left-eye image 605 and the right-eye image 606 using aknown pattern matching method. The parallax map generation circuit 514executes the following processing for each detected object.

The parallax map generation circuit 514 calculates the positional shiftamount between the midpoint which is located between the centroid of anobject image in the left-eye image 605 and that of the object image inthe right-eye image 606 and the centroid of the object image in theleft-eye image 605 as a parallax amount. In other words, the parallaxmap generation circuit 514 calculates the positional shift amount of thecentroid of an object image in the left-eye image relative to theposition of the centroid of the object image in the combined imagegenerated on the basis of the left-eye image 605 and the right-eye image606 as a parallax amount. The calculated parallax amount is a parallaxamount corresponding to the left-eye image. Of course, the parallax mapgeneration circuit 514 may also calculate the positional shift amount ofthe centroid of an object image in the right-eye image relative to theposition of the centroid of the object image in the combined image as aparallax amount corresponding to the right-eye image.

In the example shown in FIG. 11B, for the object 602, the parallax mapgeneration circuit 514 calculates a parallax amount 612, which is a halfof a parallax amount 610. For the object 604, the parallax mapgeneration circuit 514 calculates a parallax amount 613, which is a halfof a parallax amount 611. The parallax map generation circuit 514 storesinformation regarding the calculated parallax amounts 612 and 613 andinformation about the position of an image serving as a reference forthe parallax amount as a parallax map in the memory unit 508. In thisexample, information about the position of an image serving as areference for the parallax amount indicates the centroid of an objectimage in a combined image. As described above, in the example shown inFIG. 11B, there is no parallax amount for the object 603. FIG. 12 is adiagram schematically illustrating a parallax map. A parallax map 621includes a parallax amount 622 and information 623 about the position(the centroid) of an image serving as a reference for the parallaxamount.

Next, a description will be given of image reproduction processing usinga parallax map. The system control unit 509 reads out combined imagedata and parallax map data from the memory unit 508. The system controlunit 509 confirms that the position of an image serving as a referencefor the parallax amount indicated by the parallax map is the centroid ofan object image in the combined image. Then, the system control unit 509generates a left-eye image to be reproduced, that is, data for thereproduction of an image corresponding to the parallax amount, byshifting the object image included in the combined image by the parallaxamount indicated by the parallax map.

Also, the system control unit 509 inverts the sign of the parallaxamount indicated by the parallax map. The system control unit 509 setsthe parallax amount obtained by the inverted sign of the parallax amountindicated by the parallax map as the positional shift amount of thecentroid of an object image in the right-eye image relative to thecentroid of the object image in the combined image. Then, the systemcontrol unit 509 shifts the object image included in the combined imageto the extent of the inverted sign of the parallax amount indicated bythe parallax map data. In this manner, the system control unit 509generates a right-eye image to be reproduced, that is, data forreproduction of an image other than the image corresponding to theparallax amount indicated by the parallax map data.

By shifting an object image in the combined image, a pixel at a positionat which the object image is arranged is a missing pixel. Thus, forexample, the system control unit 509 imparts color space information tothe missing pixel using a known technology disclosed in Japanese PatentNo. 3524147. In other words, the system control unit 509 calculates anaverage of pixel values of pixels in the vicinity of the missing pixelas color space information, and imparts the calculated color spaceinformation to the missing pixel.

As described above, the imaging apparatus of the present embodimentcombines a left-eye image with a right-eye image to thereby generate acombined image reflecting the shape of an object. Then, the imagingapparatus generates information regarding the parallax amount on thebasis of the position of the generated combined image as a parallax map.Left-eye image data and right-eye image data, the combined image datagenerated by the image combining circuit 513, and the parallax map datagenerated by the parallax map generation circuit 514 are stored in thememory unit 508.

Then, the imaging apparatus generates left-eye image data and right-eyeimage data to be reproduced based on the combined image and the parallaxamount indicated by the parallax map. Thus, even if the shape of theleft-eye image/right-eye image obtained by the photoelectricalconversion of a light flux having passed through a different region ofan exit pupil of a photographing optical system is different from theshape of an object, an image correctly reflecting the shape of theobject can be reproduced upon reproduction of the image.

FIG. 13 is a diagram illustrating an example of a typical DCF image filestructure. A DCF image file 6000 includes a DCF header section 6003, athumbnail image section 6004, a JPEG image section 6005 and a parallaxmap section 6006.

A DCF image data section 6001 includes the DCF header section 6003, thethumbnail image section 6004, and the JPEG image section 6005. The DCFheader section 6003 is a region, in which a predetermined data size isgiven in advance, for storing DCF header information. The DCF headerinformation includes the following supplementary information and offsetinformation for specifying a region for storing each image data (Thenumber in parentheses denotes a reference numeral).

Metadata A (6007): photographing information, parameters, and the likeregarding image data stored in the JPEG image section 6005.

Offset value B (6008): offset data corresponding to the head position ofthe thumbnail image section 6004.

Offset value C (6009): offset data corresponding to the head position ofthe JPEG image section 6005.

Offset value D (6010): offset data corresponding to the head position ofa parallax map section 6006.

Each of the offset values B, C, and D is relative position informationbetween the reference position of the DCF header section 6003 and thecalculated position of each image section, and the starting position ofeach image data stored in an image file is specified using each of thesevalues.

The thumbnail image section 6004 is a region for storing thumbnail imagedata which is resized by thinning JPEG image data or the like stored inthe JPEG image section 6005. Thumbnail image data is used, for example,when a plurality of reduced images is displayed on the screen of thedisplay unit 516 (when index display is performed).

The JPEG image section 6005 is a region for storing JPEG image datawhich is obtained by compressing RAW image data processed by the imagingsignal processing circuit 506. The term “RAW image” means an imageobtained from data for each of the pixels of the imaging element, i.e.,an image prior to image processing such as development processing or thelike. The JPEG image data is data which can be handled by manyapplications. In the present embodiment, data for the combined JPEGimage of a right-eye image and a left-eye image is stored in the JPEGimage section 6005. Data for the JPEG image section 6005 can bereproduced as an image for 2D display reflecting the correct shape of anobject by additively combining both images. Note that the additioncombination is also intended to include arithmetic average processing.Furthermore, a left-eye image and a right-eye image can be restoredusing a parallax map, and thus, image data which can also be reproducedas image data for 3D display is obtained. The parallax map section 6006is a region for storing the parallax amount of image data in the JPEGimage section 6005. The parallax amount is calculated by theaforementioned method.

Next, a description will be given of an image file recording operationof the present embodiment with reference to FIG. 14. FIG. 14 is aflowchart illustrating a series of processing sequences from the startof photographing to the recording of an image file in the recordingmedium 511. Hereinafter, a description will be given of processing whena photographer makes settings for 3D photographing prior tophotographing and processing when a photographer does not make suchsettings.

In step S6101, as described with reference to FIG. 11, a series of imagecapture processing from AF (Auto Focus) processing through AE (AutomaticExposure) processing to exposure processing is performed in response tothe user operation of a release button (not shown). Next, in step S6102,processing for creating an image file including a parallax map or animage file not including a parallax map is performed, and the createdimage file is temporarily saved in the memory unit 508. A detaileddescription of DCF file creation processing in the second embodimentwill be described below with reference to FIG. 15 and FIG. 16. In stepS6103, record processing is performed by writing data of the image file,which has been generated in step S6102 and temporarily saved in thememory unit 508, to the recording medium 511.

Next, a description will be given of image file creation processing instep S6102 shown in FIG. 14 with reference to FIG. 15. Firstly, in stepS6201, the imaging signal processing circuit 506 acquires RAW dataoutput from the imaging element 505. The image combining circuit 513generates combined image data using the acquired RAW data (step S6202).The generated combined image data is stored in the memory unit 508. Instep S6203, it is determined whether or not a parallax map needs to becreated. For example, when 3D photographing is performed in accordancewith a user operation, a parallax map is required, whereas when 2Dphotographing is performed, a parallax map is not required. Here, adescription will be given of parallax map generation processing.

A description will be given of parallax map generation processing instep S6204 with reference to FIG. 16. Firstly, in step S6201, the imagecombining circuit 513 generates left-eye image data from the acquiredRAW data (step S6301), and further generates right-eye image data (stepS6302). Note that the order of generating left-eye image data andright-eye image data may be the opposite order of that described above.Resize processing for the generated two image data is performed inaccordance with the size of a JPEG image (step S6303). Next, theparallax map generation circuit 514 generates parallax map data fromleft-eye image data and right-eye image data subjected to resizeprocessing (step S6304). The generated parallax map data is stored inthe memory unit 508. Note that parallax map data may also be generatedfrom left-eye image data and right-eye image data not subjected toresize processing, and then, resize processing may be performed for theimage data.

Thumbnail image data is generated in step S6205 shown in FIG. 15, andJPEG image data is generated in step S6206 shown in FIG. 15. Processingfor reading out combined image data stored in the memory unit 508 isexecuted, and the compression/decompression circuit 515 performs imagecompression processing depending on the set mode. JPEG image data andthumbnail image data subjected to a series of processing from the startof photographing are stored in the free region of the memory unit 508.

Finally, a DCF header is created (step S6207), and processing forcreating an image file 6000 is executed using thumbnail image data, JPEGimage data, and parallax map data which are stored in the memory unit508. At this time, metadata is created from photographing informationsuch as a shutter speed, an exposure time, and the like, and thesevalues are set as the metadata A (see reference numeral 6007 shown inFIG. 13), such as photographing information, parameters, and the like,in the DCF header section 6003. Also, the offset value B (see referencenumeral 6008) corresponding to the thumbnail image section 6004, theoffset value C (see reference numeral 6009) corresponding to the JPEGimage section 6005, and the offset value D (see reference numeral 6010)corresponding to the parallax map section 6006 are set based on the sizeof data. In this manner, header information of the DCF header section6003 is generated. Thus, information (see an offset 6010) indicating thepresence/absence of the parallax map and the location thereof is storedin the DCF header section 6003. For example, if the offset value D forthe offset 6010 is a predetermined value (e.g., “0”), it means that theparallax map is not stored in the image file 6000. If a predeterminedvalue indicating that the parallax map is not included in the image file6000 is set or the offset 6010 itself is not present in the DCF headersection 6003, it can be determined that the parallax map is not present.On the other hand, if a predetermined offset value is given to theoffset value D, the parallax map is stored in the image file 6000, andthus, the storage location can be specified by the offset value D.

Next, a description will be given of image file creation processing instep S6203 shown in FIG. 15, when the parallax map is not generated,with reference to FIG. 15. Combined image data is stored in the memoryunit 508 through the processing in steps S6201 and S6202. Afterdetermination in step S6203, thumbnail image data is generated usingcombined image data (step S6205), and JPEG image data is generated usingthe same (step S6206). Unwanted usage of the memory unit 508 can beprevented by omitting parallax map generation processing.

In step S6207, metadata is created from photographing information suchas a shutter speed, an exposure time, and the like, and these values areset as the metadata A (see reference numeral 6007 shown in FIG. 7), suchas photographing information, parameters, and the like, in the DCFheader section 6003. Furthermore, the offset value B (see referencenumeral 6008) corresponding to the thumbnail image section 6004, theoffset value C (see reference numeral 6009) corresponding to the JPEGimage section 6005, and the offset value D (see reference numeral 6010)corresponding to the parallax map section 6006 are set based on the sizeof each data. Here, the offset value D to be stored in an offset 6101 isset to a predetermined value (e.g., “0”) indicating that the parallaxmap is not included in the image file 6000. Alternatively, if theparallax map is not included in the image file 6000, the offset 6010itself may not be generated.

Next, in step S6208, the image file 6000 not including the parallax mapis created on the basis of data of the regions including the DCF headersection 6003, the thumbnail image section 6004, and the JPEG imagesection 6005 which are generated in the previous steps. In the presentembodiment, a description has been given of record processing forwriting an image file to the recording medium 511, but the presentinvention is not limited thereto. The same processing as that describedabove is performed when image data or management information associatedwith image data is transferred between peripheral devices such as acomputer device, a printer, or the like.

According to the second embodiment, the combined JPEG image data of aleft-eye image and a right-eye image is stored in the JPEG image sectionof an image file, the parallax map regarding the data is enclosed in theimage file. Thus, 3D display image can be reproduced using the parallaxmap. In the application which is incapable of using a parallax map, 2Ddisplay image can be reproduced using JPEG image data subjected toaddition combination processing, and thus, the versatility is ensured.In other words, in the present embodiment, an image file which can bereproduced in both 2D display and 3D display can be created whilemaintaining a file format which is reproducible in a generally-usedapplication.

Variant Example of Second Embodiment

Next, a description will be given of a variant example of the secondembodiment of the present invention. Hereinafter, a description will begiven of the structure of an image file and image file generationprocessing, when a thumbnail image parallax map is generated, withreference to FIG. 17.

FIG. 17 is a diagram illustrating an exemplary structure of a DCF fileincluding a thumbnail image parallax map. A DCF image data section 7001includes a DCF header section 7003, a thumbnail image section 7004, anda JPEG image section 7005. The DCF header section 7003 is a region, inwhich a predetermined data size is given in advance, for storing DCFheader information. The DCF header information includes the metadata A(see reference numeral 7008) regarding photographing information,parameters, and the like, the offset value B (see reference numeral7009) corresponding to the head position of the thumbnail image section7004, and the offset value C (see reference numeral 7010) correspondingto the head position of the JPEG image section 7005. The offset value D(see reference numeral 7011) indicates offset data corresponding to thehead position of a thumbnail image parallax map section 7006. The offsetvalue E (see reference numeral 7012) indicates offset data correspondingto the head position of a JPEG image parallax map section 7007. Thestarting position of each image data stored in an image file isspecified using each of these offset values B, C, D, and E. Note thatthe arrangement order of offset values is merely an example, and mayalso be changed.

A parallax map section 7002 includes the thumbnail image parallax mapsection 7006 and the JPEG image parallax map section 7007. The thumbnailimage parallax map section 7006 stores the parallax map data generatedin accordance with the size of a thumbnail image, and the parallax mapdata is used for image reproduction processing for the thumbnail imagein 3D display. The thumbnail image parallax map section 7006 is arrangedin front of the JPEG image parallax map section 7007 (the DCF headersection 7003 side), but the arrangement order of the parallax maps inthe parallax map section 7002 may be the opposite of that shown in FIG.11. Note that the thumbnail image section 7004, the JPEG image section7005, and the JPEG image parallax map section 7007 are the same as thoseof the first embodiment, and thus, a detailed description thereof willbe omitted.

As shown in FIG. 17, the thumbnail image parallax map and the JPEG imageparallax map are arranged at the rear of the image file, resulting in animprovement in the convenience of editing such as image reproduction,parallax map deletion, or the like when the parallax map(s) is not used.

Next, a description will be given of parallax map generation processingaccording to the variant example of the second embodiment with referenceto FIG. 15 and FIG. 18. Firstly, in steps S6201 and S6202 shown in FIG.15, combined image data is stored in the memory unit 508. The imagecombining circuit 513 generates left-eye image data (step S7101 shown inFIG. 18), and generates right-eye image data (step S7102). Resizeprocessing for the generated two image data is executed in accordancewith the size of the set JPEG image (step S7103). The parallax mapgeneration circuit 514 generates a JPEG image parallax map usingright-eye image data and left-eye image data subjected to resizeprocessing for a JPEG image (step S7104).

Next, processing for resizing the left-eye image data and the right-eyeimage data generated in steps S7101 and S7102 to the thumbnail imagesize is executed so as to generate a thumbnail image parallax map (stepS7105). The parallax map generation circuit 514 generates thumbnailimage parallax map data using right-eye image data and left-eye imagedata subjected to resize processing for a thumbnail image (step S7106).Then, the DCF header section 7003 is created in the same manner as thatdescribed in the first embodiment. At this time, the image file 7000 canhave a thumbnail image parallax map and a JPEG image parallax mapseparately by providing the offset value D of the thumbnail imageparallax map section 7006 and the offset value E of the JPEG imageparallax map section 7007.

As described above, in the variant example of the second embodiment, athumbnail image parallax map is generated, and then, an image fileenclosing the map data is created. Thus, 3D display image reproductionprocessing for a thumbnail image can be performed.

Another Variant Example of Second Embodiment

Next, a description will be given of another variant example of thesecond embodiment. FIG. 19 is a diagram illustrating an exemplary filestructure when an image file including RAW image data is created.

A DCF image data section 8001 includes a DCF header section 8003, athumbnail image section 8004, a display JPEG image section 8005, and aRAW image section 8006. The DCF header section 8003 is a region, inwhich a predetermined data size is given in advance, for storing DCFheader information. The DCF header information includes metadata A (see8010) such as photographing information, parameters, and the likeregarding image data stored in the RAW image section 8006 and thefollowing offset values (The number in parentheses denotes a referencenumeral).

Offset value B (see 8011): offset data corresponding to the headposition of the thumbnail image section 8004.

Offset value C (see 8012): offset data corresponding to the headposition of the display JPEG image section 8005.

Offset value D (see 8013): offset data corresponding to the headposition of the RAW image section 8006.

Offset value E (see 8014): offset data corresponding to the headposition of a thumbnail image parallax map section 8007.

Offset value F (see 8015): offset data corresponding to the headposition of a display JPEG image parallax map section 8008.

Offset value G (see 8016): offset data corresponding to the headposition of a RAW image parallax map section 8009.

The starting position of each image data stored in an image file isspecified using the offset values B, C, D, E, F, and G.

The RAW image section 8006 is a region for storing RAW image data. TheRAW image data is data which has been read out from the imaging elementand is large-sized image data prior to image processing such asdevelopment processing, compression processing, or the like. In thepresent embodiment, a description will be given below by taking anexample of handling two types of RAW image data:

RAW image data (hereinafter referred to as “addition RAW image data”)obtained by arithmetic averaging data obtained by a plurality of PDs foreach pixel, and

RAW image data (hereinafter referred to as “non-addition RAW imagedata”) obtained by directly using data obtained by PDs in a pixelwithout performing selection processing and composition processing.

Addition RAW image data is data which is obtained by arithmeticaveraging data detected by the PDs in one pixel. Although addition RAWimage data is data subjected to calculation processing, addition RAWimage data is still RAW image data in the sense of being not suitablefor being displayed. In the present embodiment, a description will begiven of addition RAW image data as data subjected to arithmetic averageprocessing. However, data subjected to only addition processing may betreated as addition RAW image data and division processing andadjustment of the dynamic range may be performed for the resulting dataduring post processing.

A parallax map section 8002 shown in FIG. 19 includes the thumbnailimage parallax map section 8007, the display JPEG image parallax mapsection 8008, and the RAW image parallax map section 8009. The thumbnailimage parallax map section 8007 and the display JPEG image parallax mapsection 8008 are the same as those described in the variant example, andthus, a detailed description thereof will be omitted.

The RAW image parallax map section 8009 is a region for storing theparallax amount of RAW image data in the RAW image section 8006. In thecase of non-addition RAW image data, information read out from the PDsof the imaging element 505 is directly used, and thus, a parallax mapcan be restored in later processing. Thus, in the present embodiment, adescription will be given on the assumption that an image file notenclosing the RAW image parallax map section 8009 is created when RAWimage data stored in the RAW image section 8006 is non-addition RAWimage data. The thumbnail image section 8004, the display JPEG imagesection 8005, and the like are the same as those described in thevariant example, and thus, a detailed description thereof will beomitted.

Next, a description will be given of parallax map generation processingaccording to another variant example of the second embodiment withreference to FIG. 15 and FIG. 20. Firstly, in steps S6201 and S6202shown in FIG. 15, combined image data is stored in the memory unit 508.The image combining circuit 513 generates left-eye image data (stepS8101), and generates right-eye image data (step S8102). Next, thesystem control unit 509 determines whether RAW image data is eithernon-addition RAW image data or addition RAW image data to therebydetermine whether or not a RAW image parallax map needs to be created(step S8103). Whether RAW image data is either non-addition RAW imagedata or addition RAW image data may be determined by a mode or the likeset by a user or may also be automatically determined by media capacity,image size, or the like. In the case of addition RAW image data, theprocess advances to step S8104. The parallax map generation circuit 514generates RAW image parallax map data using left-eye image data andright-eye image data generated in steps S8101 and S8102, respectively(step S8104). Next, the left-eye image data and the right-eye image datagenerated in steps S8101 and S8102, respectively, are subjected toresize processing for a JPEG image (step S8105), and display JPEG imageparallax map data is created (step S8106). Furthermore, the left-eyeimage data and the right-eye image data generated in steps S8101 andS8102, respectively, are subjected to resize processing for a thumbnailimage (step S8107), and thumbnail image parallax map data is generated(step S8108).

On the other hand, if it is determined in step S8103 that RAW image datais non-addition RAW image data, RAW image parallax map data does notneed to be generated, and the process advances to step S8105. Then, JPEGimage resize processing, thumbnail image resize processing, and parallaxmap generation processing are performed. As in the DCF header section ofthe second embodiment, metadata is created from photographinginformation such as a shutter speed, an exposure time, and the like, andthese values are set as the metadata A (see reference numeral 8010 shownin FIG. 19), such as photographing information, parameters, and thelike, in the DCF header section 8003. Also, the offset values B to G areset based on the size of data, and header information of the DCF headersection 8003 is generated. In this manner, information (see the offsetG) indicating the presence/absence of a parallax map for RAW image dataand the location thereof is stored in the DCF header section 8003. Ifthe offset value G indicated by an offset 8016 is set as a predeterminedvalue (e.g., “0”), it means that RAW image parallax map data is notstored in the image file 8000. Alternatively, if the offset 8016 itselfis not present in the DCF header section 8003, it can be seen that RAWimage parallax map data is not stored in the image file 8000. If alegitimate value is given to the offset value G, it is determined thatRAW image parallax map data is stored in the image file 8000, and thus,the storage location can be specified by the offset value G.

As described above, in another variant example of the second embodiment,it is determined whether or not parallax map data needs to be enclosedin an image file depending on whether RAW image data is either additionRAW image data or non-addition RAW image data. If it is determined thatRAW image data is non-addition RAW image data, a RAW image parallax mapbecomes unnecessary, and the file capacity thereof can be reduced.

Third Embodiment

Hereinafter, a description will be given of a third embodiment of thepresent invention. An exemplary configuration of an imaging apparatus ofthe third embodiment of the present invention is the same as that of theimaging apparatus of the second embodiment, and thus, a descriptionthereof will be omitted. The imaging apparatus of the third embodimentreproduces an image reflecting the correct shape of an object using aparallax map.

FIG. 21 is a diagram illustrating an example of a typical DCF image filestructure. A DCF image file 700 includes a DCF header section 701, athumbnail image section 702, and a JPEG image section 703. The DCFheader section 701 is a region, in which a predetermined data size isgiven in advance, for storing DCF header information. The DCF headerinformation includes the following data (The number in parenthesesdenotes a reference numeral).

Metadata A (704): photographing information, parameters, and the likeregarding image data stored in the JPEG image section 703.

Offset value B (705): offset data corresponding to the head position ofthe thumbnail image section 702.

Offset value C (706): offset data corresponding to the head position ofthe JPEG image section 703.

Each of the offset values B and C is relative position informationbetween the reference position of the DCF header section 701 and thecalculated position of each image section, and the starting position ofeach image data stored in an image file is specified using each of theseoffset values.

The thumbnail image section 702 is a region for storing thumbnail imagedata which is resized by thinning JPEG image data or the like stored inthe JPEG image section 703. Thumbnail image data is used, for example,when a plurality of reduced images is displayed on the screen of thedisplay unit 516 (when index display is performed). The JPEG imagesection 703 is a region for storing JPEG image data which is obtained bycompressing RAW image data, which has been subjected to image processingby the imaging signal processing circuit 506, by thecompression/decompression circuit 515. The image data is data which canbe handled by many generally-used applications.

FIG. 22 is a diagram illustrating an exemplary structure of an imagefile when a DCF image file including left-eye image data and right-eyeimage data is created in the present embodiment. An image file 800includes a DCF image data section 801 and a left-eye/right-eye imagedata section 802.

The DCF image data section 801 is constituted by a DCF header section803, a combined thumbnail image section 804, and a combined JPEG imagesection 805. The DCF header section 803 is a region, in which apredetermined data size is given in advance, for storing DCF headerinformation. The DCF header information includes the followingsupplementary information and offset information for specifying a regionfor storing each image data (The number in parentheses denotes areference numeral).

Metadata A (810): photographing information, parameters, and the likeregarding image data stored in the combined JPEG image section 805.

Offset value B (811): offset data corresponding to the head position ofthe combined thumbnail image section 804.

Offset value C (812): offset data corresponding to the head position ofthe combined JPEG image section 805.

Offset value D (813): offset data corresponding to the head position ofa left-eye thumbnail image section 806.

Offset value E (814): offset data corresponding to the head position ofa right-eye thumbnail image section 807.

Offset value F (815): offset data corresponding to the head position ofa left-eye JPEG image section 808.

Offset value G (816): offset data corresponding to the head position ofa right-eye JPEG image section 809.

The starting position of each image data stored in an image file isspecified using the offset values B, C, D, E, F, and G.

The combined thumbnail image section 804 is a region for storingcombined thumbnail image data which is resized by thinning combined JPEGimage data or the like stored in the combined JPEG image section 805.Combined thumbnail image data is used, for example, when index displayis performed on the screen of the display unit 516. Also, the combinedJPEG image section 805 is a region for storing combined JPEG image datawhich is obtained by compressing image data obtained from the imagingelement 505, which has been processed by the imaging signal processingcircuit 506, by the compression/decompression circuit 515. The imagedata is data which can be handled by many generally-used applications.In the present embodiment, JPEG image data, which is obtained byadditively combining right-eye image data with left-eye image data bythe image combining circuit 513, is stored in the combined JPEG imagesection 805. Image data is obtained which is capable of being reproducedas a 2D display image reflecting the correct shape of an object byadditively combining the left-eye image with the right-eye image. Notethat the addition combination is also intended to include arithmeticaverage processing.

The left-eye/right-eye image data section 802 is constituted by theleft-eye thumbnail image section 806, the right-eye thumbnail imagesection 807, the left-eye JPEG image section 808, and the right-eye JPEGimage section 809. Image data which is stored in the left-eye thumbnailimage section 806 and the right-eye thumbnail image section 807 is usedwhen index display is performed on the screen of the display unit 516 sothat a thumbnail image is displayed in three-dimensional display. Also,image data which is stored in the left-eye JPEG image section 808 andthe right-eye JPEG image section 809 is used when the image data isdisplayed on the display unit 516 in three-dimensional display. Theleft-eye/right-eye image data section 802 including these image sections806 to 809 is arranged at the rear of the image file 800, resulting inan improvement in the convenience of processing performed when theleft-eye JPEG image section 808 and the right-eye JPEG image section 809are not used.

FIG. 23 is a diagram illustrating an exemplary file structure of animage file, when a DCF image file including parallax map data iscreated, according to the present embodiment. An image file 900 includesa DCF image data section 901 and a parallax map section 902. The DCFimage data section 901 is constituted by a DCF header section 903, acombined thumbnail image section 904, and a combined JPEG image section905. The DCF header section 903 is a region, in which a predetermineddata size is given in advance, for storing DCF header information. TheDCF header information includes the following data (The number inparentheses denotes a reference numeral).

Metadata A (908): photographing information, parameters, and the likeregarding image data stored in the combined JPEG image section 905.

Offset value B (909): offset data corresponding to the head position ofthe combined thumbnail image section 904.

Offset value C (910): offset data corresponding to the head position ofthe combined JPEG image section 905.

Offset value D (911): offset data corresponding to the head position ofa combined thumbnail image parallax map section 906.

Offset value E (912): offset data corresponding to the head position ofa combined JPEG image parallax map section 907.

The starting position of each image data stored in an image file isspecified using the offset values B, C, D, and E.

The parallax map section 902 is constituted by the combined thumbnailimage parallax map section 906 and the combined JPEG image parallax mapsection 907. The combined thumbnail image section 904 is a region forstoring thumbnail image data which is resized by thinning JPEG imagedata or the like stored in the combined JPEG image section 905.Thumbnail image data is used, for example, when index display isperformed on the screen of the display unit 516. The combined JPEG imagesection 905 is a region for storing combined JPEG image data which isobtained by compressing image data obtained from the imaging element505, which has been processed by the imaging signal processing circuit506, by the compression/decompression circuit 515. The image data isdata which can be handled by many generally-used applications. In thepresent embodiment, JPEG image data, which is obtained by additivelycombining right-eye image data with left-eye image data by the imagecombining circuit 513, is stored in the combined JPEG image section 905.An image is obtained which is capable of being reproduced as a 2Ddisplay image reflecting the correct shape of an object by additivelycombining the left-eye image with the right-eye image. Furthermore,left-eye image data and right-eye image data can be restored using aparallax map, and thus, image data can be reproduced as athree-dimensional display image. The combined thumbnail image parallaxmap section 906 is a region for storing the parallax map data generatedin accordance with the size of a combined thumbnail image. Also, thecombined JPEG image parallax map section 907 is a region for storing mapdata indicating the parallax amount of the combined JPEG image. Thecombined thumbnail image parallax map and the combined JPEG imageparallax map are arranged at the rear of the image file 900 as theparallax map section 902, resulting in an improvement in the convenienceof editing such as image reproduction, parallax map deletion, or thelike when the parallax map(s) is not used.

Next, a description will be given of DCF image file reproductionprocessing with reference to the flowchart shown in FIG. 24. The presentprocessing is executed under the control of the system control unit 509.In step S1001, it is determined whether or not an image replay button(not shown) provided in the operation unit of the external I/F unit 512has been operated. When it is determined that a user has not operatedthe image replay button, the determination processing in step S1001 isrepeated, whereas when it is determined that a user has operated theimage replay button, the process advances to step S1002. In step S1002,processing for reading image file data from the recording medium 511 tothe memory unit 508 via the recording medium control I/F unit 510 isexecuted. Next, in step S1003, the image file read into the memory unit508 in step S1002 is analyzed. In image file analysis processing, thestructure of the DCF image file is analyzed, which allows access to eachimage data. Next, in step S1004, it is determined whether or not theimage file is an image file for three-dimensional display based on theresult of analysis in step S1003. Determination processing for an imagefile for three-dimensional display is determination whether or not theimage file is an image file by which a left-eye image and a right-eyeimage can be prepared. In the present embodiment, it is determined thatthe image file is an image file for three-dimensional display if theimage file has left-eye image data and right-eye image data as describedin FIG. 22 or if the image file has the parallax map as described inFIG. 23. If the result of determination regarding the image file is YES,the process advances to step S1005, whereas if NO, the process advancesto step S1006.

Next, in step S1005, it is determined whether or not a stereoscopicimage reproduction setting has been set. Whether image reproduction fortwo-dimensional display or image reproduction for three-dimensionaldisplay is performed can be arbitrarily specified by a user operation.Even in the case of an image file for three-dimensional display, a usercan select image reproduction for two-dimensional display. A user canmake a desired reproduction setting by a menu selecting operation or thelike on the screen of the display unit 516 using the operation unit, andsetting information is stored in the memory unit 508. If it isdetermined in step S1005 that the setting is not a reproduction settingfor a stereoscopic image, the process advances to step S1006, and 2Ddisplay image reproduction processing is performed, that is, a 2Ddisplay image (hereinafter referred to as “2D image”) is reproduced.This processing will be described below with reference to FIG. 25. Onthe other hand, if it is determined in step S1005 that the setting is areproduction setting for a stereoscopic image, the process advances tostep S1007, and stereoscopic image reproduction processing is performed,that is, a 3D display image is reproduced. This processing will bedescribed below with reference to FIG. 26. As described above, in imagefile reproduction processing performed by the imaging apparatus of thepresent embodiment, image reproduction can be performed on both 3Ddisplay and 2D display using an image file for a stereoscopic image.

Next, a description will be given of 2D display image reproductionprocessing in step S1006 shown in FIG. 10 with reference to FIG. 25. Instep S1101, processing for acquiring 2D image data from an image file isperformed. In 2D image data acquisition processing, combined JPEG imagedata is acquired from the analyzed image file as data for beingreproduced as a 2D image. In case that a file has image data captured byother than the imaging apparatus of the present embodiment and does nothave combined JPEG image data, reproduction compatibility improves whenprocessing for acquiring another image data as a 2D image is performed.Next, in 2D image display data creation processing in step S1102,display data is created from the 2D image data acquired in step S1101.In the present processing, the display control circuit 517 processesdata decompressed by the compression/decompression circuit 515, andthus, data for being displayed as a 2D image on the display unit 516 iscreated. Next, in step S1103, the display unit 516 performs 2D displayin accordance with the display data created in step S1102 under thecontrol of the display control circuit 517. In 2D image displayprocessing, display data is processed without distinguishing between aleft-eye image and a right-eye image as in stereoscopic image displayprocessing to be described below. In this manner, a 2D image can bereproduced based on the image file.

Next, a description will be given of stereoscopic image reproductionprocessing in step S1007 shown in FIG. 10 with reference to FIG. 26. Instep S1201, it is determined whether or not a parallax map is present inthe image file. The presence/absence of a parallax map can be determinedby the result of analysis in step S1003. If it is determined that thereis no parallax map in the image file, the process advances to step S1202and left/right image 3D reproduction processing is executed. Left/rightimage 3D reproduction processing will be described below with referenceto FIG. 27. On the other hand, if it is determined that there is aparallax map in the image file, the process advances to step S1203 andparallax map 3D reproduction processing is executed. Parallax map 3Dreproduction processing will be described below with reference to FIG.28. In the 3D reproduction processing of the present embodiment, 3Dreproduction using the left/right images and 3D reproduction using theparallax map can be performed.

Next, a description will be given of left/right image 3D reproductionprocessing in step S1202 shown in FIG. 26 with reference to FIG. 27. Inleft-eye image data acquisition processing in step S1301, left-eye JPEGimage data is acquired from the image file data expanded on the memoryunit 508. Next, in display data creation processing in step S1302,left-eye image display data is created from the left-eye JPEG image dataacquired in step S1301. In left-eye image display data creationprocessing, data for displaying an image on the display unit 516 by thedisplay control circuit 517 is created based on the data decompressed bythe compression/decompression circuit 515. Next, in right-eye image dataacquisition processing in step S1303, right-eye JPEG image data isacquired from the image file data expanded on the memory unit 508. Next,in display data creation processing in step S1304, right-eye imagedisplay data is created from the right-eye JPEG image data acquired instep S1303. In right-eye image display data creation processing, datafor displaying an image on the display unit 516 by the display controlcircuit 517 is created based on the data decompressed by thecompression/decompression circuit 515. In step S1305, the display unit516 performs 3D image display processing in accordance with the left-eyeimage display data created in step S1302 and the right-eye image displaydata created in step S1304. In the present processing, image display isperformed by the known technique such that different video image lightis incident on the left eye and the right eye of a user, but adescription thereof will be omitted. In this manner, a 3D image can bereproduced based on an image file having left-eye image data andright-eye image data.

Next, a description will be given of parallax map 3D reproductionprocessing in step S1203 shown in FIG. 26 with reference to FIG. 28. Inparallax map acquisition processing in step S1401, data for a combinedJPEG image parallax map is acquired from the image file data expanded onthe memory unit 508. Next, in reference image data acquisitionprocessing in step S1402, reference image data is acquired from theimage file data expanded on the memory unit 508. The term “referenceimage” used herein refers to image data corresponding to the parallaxmap acquired in step S1401. In the present embodiment, the referenceimage is combined JPEG image data. Next, in left-eye image data creationprocessing in step S1403, left-eye image data is created from theparallax map acquired in step S1401 and the reference image dataacquired in step S1402. The parallax map includes data representing theparallax amount of the centroid of an object image in a combined imagerelative to the corresponding centroid thereof in a left-eye image.Left-eye image data is generated by moving the object image to thecentroid of the object image in the left-eye image, which corresponds tothe parallax amount, using the data. Next, in right-eye image datacreation processing in step S1404, right-eye image data is created fromthe parallax map acquired in step S1401 and the reference image dataacquired in step S1402. The present processing is performed in the samemanner as left-eye image data creation processing. In this case, theparallax amount of the centroid of an object image in the combined imagerelative to the corresponding centroid of the object image in theright-eye image is obtained by the inverted sign of the parallax amountindicated by the parallax map data. Right-eye image data can begenerated by moving the object image to the centroid of the object imagein the right-eye image, which corresponds to the parallax amount. Instep S1405, left/right image 3D reproduction processing as described inFIG. 13 is executed using the left-eye image data created in step S1403and the right-eye image data created in step S1404. At this time, colorspace information is imparted to the missing pixel. In this manner,left-eye image data and right-eye image data are generated from theimage file having parallax map data, and thus, a 3D image can bereproduced based on these two image data.

As described above, according to the present embodiment, 2D imagereproduction and 3D image reproduction can be performed using a DCFimage file. In other words, the combined image data is generated from aplurality of image data obtained by the imaging element 505, and thus,an image reflecting the correct shape of an object can be reproduced inboth 2D display using the combined image data and 3D display using thecombined image data and the parallax map data. Thus, a user can view areproduction image without an unnatural impression.

By applying a series of processing described in the present embodimentto a thumbnail image, an imaging apparatus may be provided that iscapable of reproducing a plurality of reduced images in 2D display or 3Ddisplay when index display is performed on the display unit 516. In thiscase, the image combining circuit 513 generates image data for reducedimage display from the combined image data. Also, the parallax mapgeneration circuit 514 generates parallax map data for reduced imagedisplay using the left-eye image data and the right-eye image data. Whenthe image file includes the combined image data and the parallax mapdata for reduced image display, the system control unit 509 controlsimage reproduction for three-dimensional display by generating left-eyereduced image data and right-eye reduced image data from these data.When the image file includes left-eye reduced image data and right-eyereduced image data, reduced image reproduction for three-dimensionaldisplay can be performed using these data.

Although a description has been given of the preferred embodiments ofthe present invention, the present invention is not limited to theseembodiments, but the file structure in which the arrangement order ofimage data is changed depending on various applications such as imageediting or the like may also be used. For example, when a thumbnailimage is not used for three-dimensional display, only combined imagedata may be recorded in a file without left-eye image data and right-eyeimage data, and thus, various modifications and changes may be made onthe file structure.

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiments, and by a method, the steps of whichare performed by a computer of a system or apparatus by, for example,reading out and executing a program recorded on a memory device toperform the functions of the above-described embodiments. For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. An imaging apparatus comprising: a pixel arraycomprising a plurality of pixels each of which has a plurality ofphotoelectric conversion units; one or more processors; and a memorystoring instructions which, when the instructions are executed by theone or more processors, cause the imaging apparatus to: generate a firstimage data corresponding to the plurality of pixels and a second imagedata corresponding to the plurality of pixels, wherein each signal ofthe first image data commonly corresponds to any one of the plurality ofthe photoelectric conversion units in each of the plurality of pixels,and the each signal of the second image data corresponds to theplurality of the photoelectric conversion units in each of the pluralityof pixels; and generate a single file including the first image data andthe second image data, wherein the single file comprises a thumbnailimage section that includes a thumbnail image corresponding to thesecond image data, and an image section that includes the first imagedata and the second image data, wherein signals of the first image dataand the second image data correspond to light fluxes having passedthrough different pupil regions of an imaging optical system.
 2. Theimaging apparatus according to claim 1, wherein the single filecomprises an image section, and the second image data and the firstimage data are arranged in order in the image section.
 3. The imagingapparatus according to claim 2, wherein the single file comprises aheader section, and wherein the header section includes metadata ofphotographing information and a parameter, and data of an offset amountto each image area in the image section.
 4. The imaging apparatusaccording to claim 2, wherein the second image data and the first imagedata included in the image section of the single file are a JPEG image.5. The imaging apparatus according to claim 2, wherein the image sectionof the single file includes RAW image data of the first image data. 6.The imaging apparatus according to claim 1, wherein the single file is aDesign rule for Camera File system (DCF) image file.
 7. The imagingapparatus according to claim 1, wherein the thumbnail image is generatedby compressing the second image data in the thumbnail image section. 8.The imaging apparatus according to claim 1, wherein the pixel arraycomprises a plurality of micro lenses, wherein each of the micro lensescorresponds to different pixels of the pixel array.
 9. The imagingapparatus according to claim 1, wherein the memory has furtherinstructions stored thereon which, when executed by the one or moreprocessors, cause the imaging apparatus to perform focus detection usingthe signals of the first image data.
 10. An image processing apparatuscomprising: one or more processors; and a memory storing instructionswhich, when the instructions are executed by the one or more processors,cause the image processing apparatus to: acquire an image signal outputby a pixel array comprising a plurality of pixels each of which has aplurality of photoelectric conversion units; generate a first image datacorresponding to the plurality of pixels and a second image datacorresponding to the plurality of pixels, wherein each signal of thefirst image data commonly corresponds to any one of the plurality of thephotoelectric conversion units in each of the plurality of pixels, andthe each signal of the second image data corresponds to the plurality ofthe photoelectric conversion units in each of the plurality of pixels;and generate a single file including the first image data and the secondimage data, wherein the single file comprises a thumbnail image sectionthat includes a thumbnail image corresponding to the second image data,and an image section that includes the first image data and the secondimage data, wherein signals of the first image data and the second imagedata correspond to light fluxes having passed through different pupilregions of an imaging optical system.
 11. A control method which isexecuted by an imaging apparatus that has a pixel array comprising aplurality of pixels each of which has a plurality of photoelectricconversion units, the method comprising: generating a first image datacorresponding to the plurality of pixels and a second image datacorresponding to the plurality of pixels, wherein each signal of thefirst image data commonly corresponds to any one of the plurality of thephotoelectric conversion units in each of the plurality of pixels, andthe each signal of the second image data corresponding to the pluralityof the photoelectric conversion units in each of the plurality ofpixels; and generating a single file including the first image data andthe second image data, wherein the single file comprises a thumbnailimage section that includes a thumbnail image corresponding to thesecond image data, and an image section that includes the first imagedata and the second image data, wherein signals of the first image dataand the second image data correspond to light fluxes having passedthrough different pupil regions of an imaging optical system.
 12. Anon-transitory computer-readable storage medium storing a program forcausing a computer to execute a controlling method executed in animaging apparatus that has a pixel array comprising a plurality ofpixels each of which has a plurality of photoelectric conversion units,the method comprising: generating a first image data corresponding tothe plurality of pixels and a second image data corresponding to theplurality of pixels, wherein each signal of the first image datacommonly corresponds to any one of the plurality of the photoelectricconversion units in each of the plurality of pixels, and the each signalof the second image data corresponding to the plurality of thephotoelectric conversion units in each of the plurality of pixels; andgenerating a single file including the first image data and the secondimage data, wherein the single file comprises a thumbnail image sectionthat includes a thumbnail image corresponding to the second image data,and an image section that includes the first image data and the secondimage data, wherein signals of the first image data and the second imagedata correspond to light fluxes having passed through different pupilregions of an imaging optical system.
 13. A control method which isexecuted by an imaging apparatus, the method comprising: acquiring animage signal output by a pixel array comprising a plurality of pixelseach of which has a plurality of photoelectric conversion units;generating a first image data corresponding to the plurality of pixelsand a second image data corresponding to the plurality of pixels,wherein each signal of the first image data commonly corresponds to anyone of the plurality of the photoelectric conversion units in each ofthe plurality of pixels, and the each signal of the second image datacorresponding to the plurality of the photoelectric conversion units ineach of the plurality of pixels; and generating a single file includingthe first image data and the second image data, wherein the single filecomprises a thumbnail image section that includes a thumbnail imagecorresponding to the second image data, and an image section thatincludes the first image data and the second image data, wherein signalsof the first image data and the second image data correspond to lightfluxes having passed through different pupil regions of an imagingoptical system.
 14. A non-transitory computer-readable storage mediumstoring a program for causing a computer to execute a controlling methodexecuted in an imaging apparatus, the method comprising: acquiring animage signal output by a pixel array comprising a plurality of pixelseach of which has a plurality of photoelectric conversion units;generating a first image data corresponding to the plurality of pixelsand a second image data corresponding to the plurality of pixels,wherein each signal of the first image data commonly corresponds to anyone of the plurality of the photoelectric conversion units in each ofthe plurality of pixels, and the each signal of the second image datacorresponding to the plurality of the photoelectric conversion units ineach of the plurality of pixels; and generating a single file includingthe first image data and the second image data, wherein the single filecomprises a thumbnail image section that includes a thumbnail imagecorresponding to the second image data, and an image section thatincludes the first image data and the second image data, wherein signalsof the first image data and the second image data correspond to lightfluxes having passed through different pupil regions of an imagingoptical system.